/bot run --disable-fail-fast

PR_Github #29796 [ run ] triggered by Bot. Commit: 411c254

sequenceDiagram
    autonumber
    participant Executor as Executor
    participant Sched as SimpleUnifiedScheduler
    participant Capacity as PyCapacityScheduler
    participant MicroBatch as PyMicroBatchScheduler
    participant KVCache as KVCacheManager
    participant Policy as SchedulerPolicy

    Executor->>Sched: schedule(pending_requests, running_requests, kv_cache_manager)
    activate Sched
    
    Sched->>Capacity: schedule(pending, running, kv_cache_manager)
    activate Capacity
    
    Capacity->>Policy: get_new_request_ids(pending)
    activate Policy
    Policy->>Capacity: filtered_request_ids
    deactivate Policy
    
    loop For each candidate request
        Capacity->>KVCache: find_new_context_block(unique_tokens, request)
        KVCache->>Capacity: context_block_info
        Capacity->>Capacity: fit_request_to_blocks()
    end
    
    Capacity->>Sched: scheduled_requests, paused_requests
    deactivate Capacity
    
    Sched->>MicroBatch: schedule(scheduled_requests, kv_cache_manager)
    activate MicroBatch
    
    MicroBatch->>MicroBatch: compute_chunk_sizes()
    
    rect rgb(200, 220, 255)
        note right of MicroBatch: Encoder phase
        MicroBatch->>KVCache: scheduling_has_free_blocks()
        KVCache->>MicroBatch: has_free
    end
    
    rect rgb(220, 240, 220)
        note right of MicroBatch: Context phase
        MicroBatch->>MicroBatch: select_requests_for_context()
    end
    
    rect rgb(255, 240, 200)
        note right of MicroBatch: Generation phase
        MicroBatch->>MicroBatch: select_requests_for_generation()
    end
    
    MicroBatch->>Sched: SchedulerOutput (batches, tokens)
    deactivate MicroBatch
    
    Sched->>Executor: SchedulerOutput
    deactivate Sched

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

PR_Github #31953 [ run ] completed with state SUCCESS. Commit: 82fac4d
/LLM/main/L0_MergeRequest_PR pipeline #24754 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

/bot run --disable-fail-fast

PR_Github #31985 [ run ] triggered by Bot. Commit: 82fac4d

PR_Github #31985 [ run ] completed with state SUCCESS. Commit: 82fac4d
/LLM/main/L0_MergeRequest_PR pipeline #24778 completed with status: 'SUCCESS'

Great work. I haven't look at the implementation in detail yet. Two suggestions:

1. I think we should add a simple test to CI so that we don't break the functionality by accident. E.g. run test_overlap_scheduler.py with the Python scheduler too.
2. Can we run a performance check on a smaller model like Llama-3.2-1B? The overhead should be more significant there. Ideally we should add NVTX ranges and collect nsys profiles to isolate the differences in execution time for the scheduler.

Thanks for the review! @Funatiq

1. Will do.
2. I'll run a perfmance check on Llama-3.2-1B. The time breakdown listed above is done by adding timer logs and also verified with nsys profiles.

Since you already have nsys profiles, could you report what the runtime for only the _schedule range is in both cases?

Since you already have nsys profiles, could you report what the runtime for only the _schedule range is in both cases?

Sure, this image shows the result mentioned above(nsys reports different schedule time for each iteration, so we only recorded the avgs/medians). Detailes are in: https://docs.google.com/document/d/1he4S6hzDBApMGp2Bl5PTED-hcKaRmXDZbOi9EgJlK5A/edit?tab=t.0

/bot run --disable-fail-fast

PR_Github #32429 [ run ] triggered by Bot. Commit: baeee83

/bot run --disable-fail-fast

PR_Github #32433 [ run ] triggered by Bot. Commit: baeee83

/bot run --disable-fail-fast

PR_Github #32442 [ run ] triggered by Bot. Commit: fc794cb

PR_Github #32442 [ run ] completed with state SUCCESS. Commit: fc794cb
/LLM/main/L0_MergeRequest_PR pipeline #25131 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

/bot run --disable-fail-fast

PR_Github #32456 [ run ] triggered by Bot. Commit: fc794cb

@Funatiq Hi, I added an E2E benchmark on Llama-3.2-1B as suggested. The hostoverhead also seems acceptable on Llama-3.2-1B:

• Config: GB200 + Llama-3.2-1B + Agg
• Server: ADP2 + max_batch_size 1536
• Client: max-concurrency 3072 + isl_1k_osl_1k + num_prompts 30720
• Result
  • Output token throughput has around 1.4% gap after 50runs.
  • Schedule time accounts for approximately 4.45% of host_step_time when using cpp scheduler. When using the cpp scheduler, the average per iteration host_step_time is 44.53 ms, while with the python scheduler it is 45.31 ms, an increase of 1.75%.

PR_Github #32456 [ run ] completed with state SUCCESS. Commit: fc794cb
/LLM/main/L0_MergeRequest_PR pipeline #25143 completed with status: 'SUCCESS'

Thanks for the benchmarks. Could you add a short summary to the PR description please?

Thanks for the benchmarks. Could you add a short summary to the PR description please?

Sure, I have updated the PR description. I think this PR can be merged.

@eopXD @nvpohanh Hi, could you please review/approve this PR? Thanks!